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ABSTRACT 

The Magnetospheric Multiscale (MMS) mission is a constellation of 4 observatories designed to investigate 
the fundamental plasma physics of reconnection in the Earth’s magnetosphere. The various instrument suites measure 
electric and magnetic fields, energetic particles, and plasma composition. Each spacecraft has undergone extensive 
environmental testing to prepare it for its minimum 2 year mission. In this paper, we report on the extensive thermal 
vacuum testing campaign. The testing was performed at the Naval Research Laboratory utilizing the “Big Blue” 
vacuum chamber. A total of ten thermal vacuum tests were performed, including two chamber certifications, three dry 
runs, and five tests of the individual MMS observatories. During the test, the observatories were enclosed in a thermal 
enclosure known as the “hamster cage”. The enclosure allowed for a detailed thermal control of various observatory 
zone, but at the same time, imposed additional contamination and system performance requirements. The environment 
inside the enclosure and the vacuum chamber was actively monitored by several QCMs, RGA, and up to 18 ion 
gauges. Each spacecraft underwent a bakeout phase, which was followed by 4 thermal cycles. Unique aspects of the 
TV campaign included slow pump downs with a partial represses, thruster firings. Helium identification, and 
monitoring pressure spikes with ion gauges. Selected data from these TV tests is presented along with lessons learned. 

INTRODUCTION 

The Magnetospheric Multiscale (MMS) mission is a constellation of four satellites that will use Earth’s 
magnetosphere as a laboratory for studying fundamental plasma processes of magnetic reconnection, energetic particle 
acceleration, and plasma turbulence. These processes play an important role in space weather. Each of the four 
identical observatories was built and integrated at the NASA Goddard Space Flight Center. In order to prepare for 
the minimum 2 years long mission, each observatory (OBS) has undergone an extensive environmental testing 
campaign. This testing included thermal vacuum testing, which is the subject of this paper. As will be discussed in 
more detail later, the observatories were tested individually in a large vacuum chamber at the Naval Research 
Laboratory in Washington, D.Ct. Pressure and cleanliness requirements imposed by the instrument suite as well as a 
suite of special tests, such as thruster firings, imposed a number of contamination control (CC) challenges. The 
mitigation strategies as well as lessons learned are the subject of this paper. 

Figure 1 Figure 1 shows the observatory in a NASA/GSFC clean room. This picture was taken during 
preparation for an acoustic testing, in which the instruments were covered with soft covers, and a Llumalloy drape 
(not on yet) is to be suspended around the sides of the observatory. The crossed fishing lines in front of the solar array 
were in place to prevent the drape from contacting the observatory. From this picture, we can see the configuration of 
the MMS observatories. Each observatory consists of an octagonal bus divided into two sections. The bottom section 
(spacecraft deck) contains the solar arrays, while the upper section is the instrument deck. A small subset of 
instruments was also located below the spacecraft deck. The observatories also contains several deployable 
magnetometer and electric field booms as well as antennas. 


Instrument Suite 

The instrument suites (IS) on each observatory consists of 27 instruments for measuring plasma composition, 
fluxes of energetic particles, and electric fields. Majority of instruments utilize some combination of micro-channel 
plates (MCP) and solid state detectors (SSD) to magnify and collect the incoming signal. The presence of these two 
technologies makes the detectors and the charged particle focusing “optics” sensitive to particulate and molecular 
contamination. Instruments utilizing micro-channel plates (MCP) are sensitive to molecular contamination, moisture, 
and conductive particles. Instruments with solid state detectors (SSD) are sensitive to moisture. See Figure 2 for IS 
locations on MMS Observatory. The MMS contamination control team came up with various mitigations to protect 
the IS during TV testing that would also afford opportunities to collect pertinent test data to validate the molecular 
cleanliness of each observatory. 



Figure 1. MMS Observatory in NASA/GSFC cleanroom 
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Figure 2. IS layout on IS deck, seen from underneath 



Thermal Vacuum Configuration 

The TV test consisted of an initial bakeout phase, followed by four hot and four cold cycles. The test 
concluded with a short decontamination phase prior the chamber repress. Figure 3 Figure 3 shows the typical MMS 
thermal vacuum profile. The thermal testing was conducted in the “Big Blue” thermal vacuum chamber at the Naval 
Research Laboratory (NRL). The project considered NRL because of unavailability of the GSFC’s large Space 
Environmental Simulator (SES) chamber due to James Web Space Telescope (JWST) occupancy. Therefore the 
project pursued and actively helped get the Big Blue chamber and surrounding area ready for MMS. The chamber 
had undergone renovation prior to MMS showing up. These renovations included installation of new thermal shrouds 
and replacement of at least one cryopump. 



In order to achieve precise thermal control, the observatories were installed within a thermal enclosure 
colloquially known as the “hamster cage”. Two identical hamster cages were built. This allowed for one observatory 
to be prepared for the test while another one was being testing inside the vacuum chamber. Each hamster cage (HC) 
consisted of 40 cryopanels in 6 thermal zones, which can be seen in Figure 4 Figure 4 . The project procured 4 thermal 
condition units (TCU) to control the zone temperatures by pushing heated or cooled gaseous nitrogen (GN2) through 
the plumping lines. The remaining two zones were controlled by omega controllers (OM). The horizontally loaded 
“Big Blue” NRL chamber is roughly 16 feet in diameter and 30 feet long. The hamster cage was made to fit, with it 
being roughly 13 feet wide and 10 feet high. The Hamster cage setup required 90+ power supplies to provide active 
control to all the thermal circuits. The project had to procure, clean, and bake out hundreds of feet of cryogenic-rated 
stainless steel braided flexible hose of various diameters, for all the internal connections from the headers to the 
hamster cage connections. 

The IS team also requested ion gauges to test the DES, DIS, and HPCA charged particle sensors. Therefore, 
the hamster cage had to be designed with at least 9 ion gauges (4 for DES, 4 for DIS, and one for HPCA). There were 
also plentiful electrical harnessing that was used for communication to the spacecraft and instruments. Of course any 
of this testing support equipment that contained polymeric material had to be prebaked in numerous bakeouts at GSFC 
and Applied Physics Laboratory (APL) TV chambers. NRL didn’t have large enough clean tents, so the project 
specified and procured a very large cleanroom that would allow for handling the preparations of two observatories 
and two hamster cages in parallel, all in front of the chamber. 
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Figure 4. a) Hamster cage structure and b) cryopanel designations 

CHAMBER READINESS 

There were 2 clean dry empty bakeouts or precerts of the chamber before any Hamster cage bakeout 
proceeded. We (project personnel) wanted to verify that the shrouds did not leak when going hot or cold and check 
the pumping performance of the 3 of the cryopumps. The cryopumps have a 35” diameter and an advertised capability 
of each pumping 30,000 liters/sec of gaseous nitrogen (N 2 ). The cryopumps are typically cooled to <15°K, which is 
cold enough for rapid pumping of water vapor, N 2 , and oxygen (O 2 ), but not necessarily Helium (He). In addition the 
chamber shrouds was freshly painted with a low outgassing flat black top coat and though said to have been pre-cured, 
it was strongly desired to verify that with a high temperature bakeout with minimal test support hardware in the 
chamber. 


Cryopump pumping performance was also derived from this precert testing. There was a 33% loss in 
pumping capability going from 2 to 1 cryopumps and only a 15% gain going from 2 to 3 cryopumps. The shrouds 
were exercised hot, to 100°C, and cold, flooded with liquid nitrogen (LN 2 ), to verify no leaks. TQCM data yielded a 
low outgassing rate for the Big Blue chamber with respect to condensables on a TQCM set to -20°C, while pressure 
levels obtained a low and steady 2E-7 torr at 50°C. Chemical analysis from rinsate of the facility scavenger plate, 
termed the Contam plate, yielded low levels of hydrocarbons and various plasticizers. Thus the chamber was deemed 
ready to precede with MMS hardware pretesting. 

Subsequent hamster cage bakeouts were termed Dry Runs and would be limited to 60°C mainly due to the 
gold iridited finish on the aluminum framing. The hamster cage cryopanels were coated with an easy-to-clean, non- 
outgassing black anodization on both sides. Cyrogenic rated D-shaped tubing was stitched welded to the outer surface 
for fluid transfers. Strip heaters, along with heat dispersing silver plated copper braid, were affixed about every foot 
in between the tubing. These higher watt density heaters were used to trim the temperatures of the cryopanels during 
most test phases by adjustments to the power supplies. All parts and fasteners of the hamster cage were ultrasonically 
precleaned by the contamination control staff. Cryopanels and framing just received IPA wetted wipe offs. Assembly 
of both cages was completed at NRL in clean tents (holding Class lOOK conditions). A low outgassing Braycote was 
used on the million fasteners used to hold the cages together. 

To complete the thermal hookup from the hamster cage to the 4 new thermal conditioning units (TCUs) 
required hundreds of feet of cryogenic-rated flexible stainless steel all metal hoses. All lines and 2” diameter headers 
were ultrasonically bathed and prebaked out to 100°C before being used in the Big Blue chamber. A completed 
hamster cage, with cryogenic flex line jumpers attached, weighed almost 5000 pounds. A MMS Observatory ready 
for testing weighed 2100 pounds. TV test readiness involved lifting the OBS (by overhead crane) and placing it onto 
the Hamster cage base. The spider shaped hamster cage framing, minus side cryopanels, was then lifted over the top 
of the spacecraft. Solar Array panel covers and IS Melinex (clear poly) covers and all red tag covers were removed 
as the side cryopanels were installed around the spacecraft. The vapor deposited aluminum (VDA) Mylar 


contamination (contam) barriers were installed during the cryopanel installation. They were requested by 
contamination control to reduce possible silicone transmission from the solar arrays back to the IS “optics”. The 
barriers provided a thermal break as well, since the IS were to operate at different temperatures than the solar arrays. 

Once fully assembled the same overhead crane was used to lift the entire assembly up and placed onto a 
chamber sled that was on an air barge. This barge was then able to be positioned in front of the vacuum chamber. 
The assembly was then staged here, just outside the chamber while electrical and thermal hookups took place and all 
the remaining cryo-flex lines were attached to fittings to cryopanels on the Hamster cage. Once the spacecraft was 
hooked up and put through an abbreviated aliveness verification and thermal completed their checkout, the chamber 
was cleaned back to front to remove accumulated particulates during the buildup. The sled with Hamster cage and 
OBS was then winched into the chamber. After a few more quick checkouts, then the chamber door was closed, the 
IS purge was stopped, and pump down would begin. 

MONITORING EQUIPMENT 

The IS micro-channel plates and solid state detectors are contamination sensitive to molecular and particulate 
contamination. To assist with monitoring contamination real-time events happening inside the chamber, there were 
several devices used. Each Hamster cage thermal enclosure was outfitted to hold four (4) thermoelectric-controlled 
Quartz Crystal Microbalances (TQCMs), which was essentially the maximum available due to conflate limitations. 
The number of QCMs was adequate to track outgassing from within the hamster cage and from the OBS. Fifteen (15) 
Ion gauges (IG) were attached to the enclosure to monitor pressure levels and to provide ions to IS. Another IG was 
attached to a chamber conflate that allowed for pressure comparison with 2 different facility pressure gauges. The 
facility pressure gauges included a low pressure gauge which was especially helpful for tracking real-time pressure 
during pump_downs and ventbacks. A residual gas analyzer (RGA) was affixed to the chamber shroud that allowed 
for monitoring individual gases up to an atomic mass unit (amu) of 200 being released inside the chamber. 

There were also scavenger plates that helped control outgassed molecules inside the chamber. There was a 
6” by 18” scavenger Plate affixed to IS cryopanel Bay 2, thermally isolated from the Hamster cage with 2” long Delrin 
standoffs, to collect outgassed material from the OBS bus vent in Bay 2. There was a special duct affixed to the 
cryopanel and extended over the OBS vent, to assist with passage of outgassed molecular mass to the SP. The very 
large facility Contam plate was hung in the back of the chamber. The backside of the plate facing the chamber shroud 
was blanketed. There were also 2 to 4 “witness” aluminum foils placed inside the Hamster cage on the bottom deck 
cryopanels just prior door closure to collect outgassed non-volatile residue (NVR) throughout TV testing. These foils 
were collected post-test and chemically analyzed along with rinsate from the scavenger plates. 

The TQCMs were attached to the Hamster cage, facing inwards to view outgassed molecules through 2” 
diameter holes. Each were mounted on copper heat sinks and were able to be individually thermally controlled. Table 
1 notes the QCM locations. 


Table 1: QCM locations 


TQCM# 

Location 

Purpose 

1 

Bay 2 IS cryopanel, off 
centered 

Monitor outgassing around vent 
area and IS in Bay 2 

2 

Bay 3 IS cryopanel, centered 

Monitor outgassing from 
DIS2/DES2 in Bay 3 

3 

Bay 6 IS cryopanel, centered 

Monitor outgassing from HPCA 
in Bay 6 

4 

Bay 6 SA cryopanel, centered 

Monitor outgassing from Solar 
Array in Bay 6 


The IS team had requirements to test the functionality of their instruments in vacuum and they needed an ion 
source. Therefore each instrument had a dedicated ion gauge on the Hamster cage enclosure. These micro ion gauges 


were offset from the respective IS and mounted to the top deck cryopanels. Table 2 notes the Ion Gauge locations. 
Note, these IG had similar specifications as commonly used by the Fast Plasma Investigation (FPI) DBS and DIS 
instruments during their assembly and test at the instrument level. Small rectangular holes in the cryopanels allowed 
for ion throughput to IS. This gauges had dual functionality in that they could be ion source and also monitor pressure 
levels. Pretesting of several ion gauges provided confirmation to thermal operating ranges and functionality of heaters 
that were attached to the housings. 


Table 2: Ion Gauge Locations 


Ion Gauge # 

Location 

Purpose 

I 

Outside HC, IS cryopanel Bay 3 

External HC pressure 

2 

Outside HC, IS cryopanel Bay 7 

External HC pressure 

3 

Inside HC, between IS cryopanels Bays 1 & 2 

Internal HC pressure for IS 

4 

Inside HC, between IS cryopanels Bays 3 &4 

Internal HC pressure for IS 

5 

Inside HC, between IS cryopanels Bay 5 &6 

Internal HC pressure for IS 

6 

Inside HC, between IS cryopanels Bay 7 & 8 

Internal HC pressure for IS 

7 

Outside HC, Top cryopanel Bay 1 

Ion Source for DIS I + External HC pressure 

8 

Outside HC, Top cryopanel Bay 3 

Ion Source for DIS2 + External HC pressure 

9 

Outside HC, Top cryopanel Bay 5 

Ion Source for DIS3 + External HC pressure 

10 

Outside HC, Top cryopanel Bay 7 

Ion Source for DIS4 + External HC pressure 

II 

Outside HC, Top cryopanel Bay 1 

Ion Source for DES I + External HC pressure 

12 

Outside HC, Top cryopanel Bay 3 

Ion Source for DES2 + External HC pressure 

13 

Outside HC, Top cryopanel Bay 5 

Ion Source for DES3 + External HC pressure 

14 

Outside HC, Top cryopanel Bay 7 

Ion Source for DES4 + External HC pressure 

15 

Outside HC, Top cryopanel Bay 6 

Ion Source for HPCA + External HC pressure 

16 

On Chamber Shroud, right side 

Chamber pressure comparison 


TEST PLAN PROFILE 

The test plan had CC inputs embedded to protect the OBS during all phases of vacuum exposure. After the 
IS purge was stopped and manually capped at chamber conflate, the pump down of the chamber could begin. A slow 
pump down was requested to reduce disturbing small particles inside the chamber and to lessen vibrations over 
instruments with MCPs. Several slow represses were conducted to help remove water vapor. After a couple cycles 
the chamber cyropumps were enabled and the test proceeded to high vacuum. Additional vacuum pumping time 
transgressed to continue evacuating water vapor before thermal transitioning to bakeout settings would occur. 

Once the TCU checkouts were completed, the facility and HC scavenger plate were flooded. The QCMs and 
RGA were activated as well. When the pressure was low enough, several Ion Gauges were also turned on. Once 
chamber pressure levels lowered adequately, the hamster cage and thermal shrouds were stair-stepped up to bakeout 
settings. The bakeout settings were be in force until QCM readings had achieved the goals with margin and vacuum 
pressure (mainly driven by water vapor) had sufficiently dropped. Thermal had to reduce the bottom cryopanel during 
the bakeout in order to keep larger internal OBS subsystems below maximum temperatures. For example, the battery 
had a red limit of 35°C. Internal OBS and IS operation heaters helped also transition and maintain payload 
temperatures at bakeout settings. The cooling of this bottom cryopanel created a molecular collection surface that 
required additional bakeout time before proceeding to the next phase. Thruster firings and catalytic bed warmup also 
provided another reason to continue bakeout settings for several hours. 

Thereafter the OBS was to begin the majority of the comprehensive performance tests (CPT); refer back to 
Figure 3. There were 4 hot and cold cycles to complete. High voltage testing of the IS during hot and cold plateaus 
wa splateaus was completed when internal hamster cage pressures were <lE-6 torr. The A and B side of electronic 
boxes had to be powered on and complete similar CPTs during these plateaus as well. After another Magnetic Boom 
partial deployment and another thruster firing in the last cold plateau, the OBS was to be warmed back up to ambient 


temperature. As in the bakeout, the return to ambient the OBS was warmed up in a stair-stepped method in a way to 
keep the spacecraft always warming than the surroundings. The entire test profile was around 25 days, from pump 
down to vent back. OBS2 was the first spacecraft to be TV tested and it had a slightly different thermal testing profile. 
It underwent thermal balance testing first and then was vented back. After a few reconfigurations were made it was 
then back under vacuum to complete 3 additional thermal cycles. 

DATA ACQUISTION 

Thermal vacuum operations at NRL were overseen by a small GSFC support team that included a 
contamination control engineer, 2 thermal engineers, a quality assurance, an electrical ground support person, and a 
project management representative. There were at least 2 NRL thermal vacuum chamber operators at all times, in 
addition to NRL management support during the day. The main OBS and IS support teams remained back at GSFC. 
Verbal communications between the centers were enabled by voice loops. Test data from the chamber that included 
300 thermal couples, ion gauge pressures, TQCMs were collected by a NRL-developed program called CDACS 
(Computerized Data Acquisition and Control System). The Python-based GUI control system allowed for creating 
real-time plots of all areas of interest. To see the data back at GSFC required a secure Thin Client internet connection 
between the centers. GSFC utilized their ASIST (Advanced Spacecraft Integration and System Test) program to 
acquire IS and OBS telemetry. System telemetry data from the OBS could then be set up in tabular displays and 
options in the program made it possible to create instantaneous plots of real-time telemetry. 

The RGA screen, the Contamination Control console screen, and the Systems console screen at NRL were 
mirrored and those images and chamber raw data were accessible by internet connection. CDACS data and ASIST 
telemetry were updated every minute. This then afforded all test personnel to see console data from anywhere they 
had internet access. Figure 5 shows a picture of the typical console data. Camera/screen options are noted in the left 
side. Having all the data streamed in the same data acquisition system greatly simplified anomaly investigations, 
which seemed to occur during every TV test. CDACS data plots could be set to as short as 15 minutes and at times 
they were set to 24 hours. 



Figure 5. CDACS data plots seen in Thin Client feed from NRL 














TEST RESULTS 


Multiple pump downs and represses were requested to be as slow as possible to reduce particulate stir up and 
to reduce vibrations to MCPs. These multiple cycles helped remove water vapor, thus lessening the work the 
cryopumps would have to do later. The chamber utilized rough pumping with a small blower until a pressure of <8 
torr was reached and then larger blowers were used to obtained pressure levels <0.05 torr (termed 50 microns). Once 
pressure levels were < 0.05 torr, valves were closed and a ball valve was adjusted to back fill the chamber with GN2 
at slow rates to 500 torr. The rough pump valves were then closed and the chamber was then backfilled with GN 2 at 
slow rates of 50 torr/hr to 200 torr and then 100 torr/hr to 500 torr. Another couple cycles were performed. Each 
cycle took about 9 hours. Typically after the 3^^ pump down, the chamber was “crossed over”, which involved opening 
the cryopumps and proceeding to high vacuum and conducting thermal TCU checkouts. See Figure 6 for a typical 
multiple pump down and repress as monitored with the low pressure gauge. 


OBSl Pumpdown/Repress 



Figure 6. Typical OBS cycle purge 


The Bakeout phase was important for removing additional water vapor and polymeric material from the OBS 
and from the surrounding GSE. Water vapor outgassing was known to suppress obtainment of low vacuum pressures 
and the high temperature bakeout of the OBS to 50°C enabled reducing water vapor from the environment at a much 
quicker rate. The ultimate goal of driving off the water vapor was for obtaining internal Hamster cage pressures of < 
lE-6 torr so the IS could conduct high voltage operations. Since the empty Big Blue chamber could easily get to the 
low E-7 torr range by normal pumping with 2 cryopumps, the OBS test setup required a decent bakeout to achieve 
desired pressure levels later on. CC monitored ion gauges and the RGA (see Figures 7 & 9) for decreasing pressures 
mainly due from water vapor and light molecular weight polymeric material. QCMs were set to -20°C during this 
phase and the delta frequency data were desired to meet pre-test goals of around 200 Hz/hr (see Figure 8) which were 
computed based on obtaining values an order of magnitude lower than required. Almost all MMS flight subsystems 
and IS were individually thermal vacuumed baked out to temperatures of 50°C or higher before integration onto the 
spacecraft and their respective certification outgassing rates were always much lower than required. So the fully 
assembled OBS was expected to exhibit low outgassing rates and come closer to the respective QCM delta frequency 
goals. 
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Figure 7 . Pressure decay inside and outside of Hamster cage during bakeout 


Delta Frequency response 









Figure 8. QCM Delta Frequency decay during bakeout mode 


RGA scan in OBSl Bakeout phase 



-cn,thsnC02 

P-MKS 


1.0C-0S 

j/uVH4iKOti 1/n^HCtoq Vii/iACicOJ i/n/14 i/ii/ijuma i/l:f/|4aa3 j/W/l-Ki;cn 


Figure 9. RGA partial pressures during bakeout mode 


The thruster firings was an exciting time to validate the MMS thruster performance in vacuum. Each of the 
12 thrusters were to be individually fired for 50 milliseconds. The propulsion tanks were pre-filled GN 2 (28 amu) with 
an Argon tracer (40 amu). Propulsion agreed to use GN 2 , instead of Helium, because the Big Blue chamber did not 
have a turbo pump and GN 2 (w/ Argon) could still be traced (by an RGA) and could be pumped from the chamber with 
the cryopumps. Helium would not have been pumped out effectively with the chamber’s cryopumps. If Helium had 
been used the project would have had to procure a pricey turbo pump for the chamber or add support time at NRL in 
all TV exposures because the chamber would have had backfilled to ambient atmosphere to remove the Helium 
buildup. Therefore using GN 2 (w/a tracer) made the most sense. 

Prior to thruster firing the catalytic (cat) beds were activated to test their response. Polymeric material nearby 
outgassed briefly as the 4 axial thruster catbeds approached 220°C and the 8 radial housings approached 120°C. The 
outgassing “bumps” are seen in Figures 7-9 as well. QCMs were set to -45°C during this phase which was to emulate 
the coldest operating range of the IS. Figure 7 shows that the outgassing load is roughly 2.5X the rate at -20°C. Most 
of this gain was from the entire OBS outgassing, not just from materials around the thruster catbeds. For RGA 
monitoring of the thruster firings, the on-shift CC person switched the scan mode to pressure versus time and shortened 
the scan time to 3 seconds. Once prop initiated a firing, it only took a few seconds to verify the on-screen RGA 
response. Other atmospheric gases were continued to be tracked for any buildup during the process, which included 
Helium (4 amu), water vapor (18 amu), and Carbon dioxide (44 amu). 

The thruster firings were conducted in the same order every time, starting with the axial thrusters 9, 10, 11, 
and 12 and then going 1 through 8 of the radial thrusters. Internal IGs were also used to track the internal Hamster 
cage pressures. The instrument suite were always placed in safe mode prior to thruster firings, so there was no voltage 
issues to worry about during this activity. The internal IGs pressure levels exceeded lE-5 torr and sometimes shut off 
when several upper radial thrusters were fired and required resetting (turning them back on before next firing). Figure 
10 shows a typical Ion Gauge response during thruster firings. As one can see, vacuum pressure levels inside the 
chamber, but outside the Hamster cage (refer to IG #1, 2, & 16) increased a little bit, but not to the point of flooding 
the cryopumps. As a safety precaution, thruster firings were always done with 2 cryopumps open and the 3^^ pump 
was idle, but ready, just in case pressure levels caused a significant pressure increase to trigger their gate valves to 
close. 



Ion Gauge Reaction During Thruster Firing A&B-side Bakeout Mode 



Figure 11. OB SI ion gauge response during thruster firing 


Transitions between hot and cold plateaus typically resulted in the discovery of GN2 leaks or water vapor 
outgassing, which resulted in pressure spikes. The IS team typically placed the IS trans safe mode when thermal 
transitions occurred. The most occurrences pressure spikes happened early in TV. After the transitions had occurred 
a few times, the spikes seemed to reduce. Sometimes the reduction in spikes were because the TCUs were controlled 
better during the transitions and it also appeared at times that the leaks sealed themselves! The actual source of the 



GN 2 leaks varied as well, since the flex lines had to be disconnected between TV tests and 2 different Hamster cages 
were used. During TV operations leak checking with Helium and later with Argon helped in few instances but not in 
all. Monitoring TCU trends helped is several other instances, and then other instances we could not identify the source 
of the leaks. We identified that TCU #3, supplying GN 2 to solar array cryopanels did exhibit trends that earmarked it 
as a source of leaks, especially during OBS #3 TV. The leak could have been anywhere along the GN 2 supply or 
return circuit. Early on in OBS TV testing pressure spikes were associated with water vapor being released from cold 
spots on cryopanels or from cryo-flex lines that may have warmed up to greater than -1 10°C. Therefore, thermal was 
requested to minimize the coldest inlet temperature to cryopanels at -90°C, and the set point for scavenger plates was 
to < -150°C therefore leaving some buffer to keep the supply lines and panels away from the magical -110°C 
temperature. These pressure spikes kept the support team glued to the console screens, especially when the IS were 
powered on and they were not in trans safe mode. Hot to cold transitions took almost 20 hours, while cold to hot went 
in half the time. Figure 12 shows an especially spikey time on 01/15/2014 when OBSl was in transition from hot 
plateau #1 to cold plateau #1. 
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Figure 12. Pressure spikes during OBSl transition 

Subsequent transitions were less eventful and OBSl was able to conduct HV operations later in testing as the 
chamber was able to hold < lE-6 torr for majority of the remainder of TV operations. See Figure 13 for the entire 
vacuum pressure profile. Really only during thruster firings, conducted at the end of Cold plateau #4, did the internal 
HC pressure levels exceed lE-6 torr, which was anticipated. 
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Figure 13. OBSl TV pressure profile 


At the end of Cold Plateau #4, TV testing on the IS and most subsystems was completed and thus the OBS 
and GSE were stair-stepped up back to ambient temperature. The OBS and IS surfaces were actually stepped up prior 
to the respective cryopanels. OBS and IS operational heaters were activated and prewarming occurred before 
cryopanels. The thermal shrouds (TS) lagged every other surface in the chamber (except SP) and they were typically 
only warmed to 20°C. So the thermal shrouds were typically a source of outgassing for the next OBS test and as such 
precautions were taken then, as mentioned, it was warmed up to lag the OBS and IS as well when going hot. Figure 
14 shows a typical cryopanel warmup along with pressure response by a facility gauge and an Ion gauge (#2). Because 
of the slow backfill, it became apparent to keep the Cryopanels and shrouds on, maintained at +20°C to minimize 
future condensation during ventback. 

Cryopanels & Shrouds During OBSl Ventback 
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Figure 14. Cryopanel stair-stepped warmup and pressure response 


OBS3 TV ISSUES 

OBS3 TV provided the most entertainment for the project support staff. This testing was delayed from 
following OBS4 by almost 3 months. During that dwell timeframe, the project desired to conduct vibration on the 
stacked observatories. So in the meantime, an eventful third Dry Run was performed on the hamster cage to be used 
for OB S3. The belts on the main rough pump blowers broke during pump down. The Contam plate thermocouple 
leads were not connected inside the chamber. And lastly the cryo-flex lines inside the chamber were not hooked up, 
so the TCUs were not checked out during the bakeout, therefore there was no cold cycle phase in this dry run. Work- 
arounds enabled completion of a satisfactory 2 day bakeout. Another in-line blower was used to rough pump the 
chamber, while new belts were ordered. A spare thermocouple was attached to the Contam plate supply line, external 
to the chamber, which essentially kept the Contam plate flooded with LN 2 during vacuum. The TCU checkout was 
always a part of the OBS TV plan, so that formal checkout would have to wait until OB S3 TV. 

OB S3 wanted to get underway before the July 4*^ weekend and there was a flurry of activity the week before 
the Friday’s pump down. All internal cryo-flex lines were again leaked checked (with Helium) after they were 


connected. Leak checking the TCUs circuits from outside the chamber before TV was actually requested and 
completed. No leaks were noted from these checkouts. The belts on the rough pumps were replaced and the pumps 
were activated for a few hours to validate operation. The Contam plate internal TC harness was checked to have been 
connected. All other checkouts proceeded nominally. 

While the pump down and represses proceeded the TCUs were activated. TCU#3 trips a circuit break when 
the blower was activated setting off NRL console alarms. It was discovered a manual valve on the TCU from a purge 
line was left open, during the pre-TV leak checking. This was foreshadowing for TCU#3 issues! The test proceeded 
to high vacuum. Initial activation of the RGA created some issues and after the external controller was swapped out, 
the RGA resumes working nominally, but we (CC) did lose several days of early data gathering of gas load inside the 
chamber. TCU and omega checkouts on cryopanels, once under vacuum, also provided more excitement. A solenoid 
valve for Omega 10, which controls the Bottom Ring, was stuck open and these cryopanels got to -180°C for short 
period (1 hour). It became another scavenger plate during this short duration. The only viable solution, at the time, 
was to stop all LN 2 flow, which was to affect the Contam Plate and HC scav plate as well. The scav plates warmed 
up briefly, which created a large water vapor induced pressure spike to lE-4 torr. The solenoid was fixed and bottom 
ring cryopanel was warmed back up to >30°C. The QCM and RGA computer were discovered to be miswired, as 
they had been reversed for another TV test in another chamber recently brought on-line. This delayed QCM data 
tracking for the initial 12 hours of bakeout. Fortunately internal HC pressure data was able to be monitored by several 
ion gauges. Once the RGA was brought back on-line, the support staff noted elevated partial pressure levels of Helium 
in the chamber, which were higher than water vapor! At this point, it was agreed upon to regenerate the cryopumps 
to reduce Helium levels, this had worked in the past with some success. All 3 cryopumps were cycled through for 
regeneration and no change in Helium levels were noted. Thruster firings commenced and no increase in Helium 
levels was detected. The bottom cryopanel were then warmed for several hours, as had been conducted for all previous 
TV tests. Once the QCM delta frequency data had reduced to an acceptable level, transition to Hot Plateau 
commenced. 

Thermal set points on the TCU #3 were inadvertently set to a temperature rate of 5°C/min and this created a 
pressure issue inside the TCU. LN 2 was being sent into the internal cryo-flex lines into a warm chamber (Shrouds 
still at 40C), which caused the LN 2 to flash to a gas inside the lines. The gas expanded so quickly that it burst the 
pressure relief disk on the external TCU. The TCU settings were reset to change at rate of UC/min and the burst disk 
was replaced, issue resolved. The RGA noted large amounts of water vapor and carbon dioxide spikes during the 
vent. Helium and Nitrogen levels were still at elevated levels and exhibited no changes as the cryopumps were regened 
one more time each. Overall chamber pressure levels were high, around 6 E-6 torr. OBS CPTs continues, but some 
IS related CPTs were bypassed (for later completion) due to the elevated pressures. The TCUs were thought to be 
part of the pressure spikes so their temperature set points were incrementally warmed from a low of -90°C to -85°C, 
to alleviate cold spots on the cryopanels. The project decided to open all 3 cryopumps. This did little to reduce overall 
Helium levels, so it was reluctantly decided to vent chamber back to 100 torr, in a hopeful exercise to try and rid the 
chamber of this “excess” Helium! The return to ambient procedure was followed, the chamber was backfilled to 100 
torr and then pumped back down to high vacuum. Back under high vacuum at ambient temperature and the best the 
chamber can get is 4E-6 torr. Helium is still the highest partial pressure and close to pre-backfill levels! 

Belief that leaks in the systems still are there, leak checking with Argon was requested. Other than a spotty, 
small leak on TCU#3, nothing else was clearly identified. The project decides to put the cryopumps through a lengthy 
regeneration process that involves cycle purging the cryopump volume with GN 2 10 times. Each cycle purge is about 
an hour and it took about 18-20 hours to complete a cryopump regeneration. There was no noticeable decrease in 
Helium levels after this exercise. Based on discussions with GSFC and APE vacuum personnel, the Project decides 
to ventback the chamber to atmospheric pressure and install a turbo pump on the one remaining conflate on the Big 
Blue chamber. The turbo pump was a cobbled together setup that was to assist with pumping Helium out from the 
chamber, if it should still be predominately present. During the non-vacuum time, the TCUs were again pressurized, 
but there was no noted pressure losses in the circuits. The Turbo pump was installed, checked out on for on/off 


operation and pumped to the manual gate valve. Once deemed safe, the chamber underwent 1 pumpdown/repress 
cycle and went to high vacuum. 

The RGA was reactivated and now finally Helium levels were low, like in previous TV tests, and were 2-3 
orders of magnitude lower than water vapor. Now the chamber is exhibiting elevated partial pressures from N 2 and 
water vapor. The turbo pump was turned on for a short duration to see if it made any difference in Helium or any 
other gas. No noticeable change after several hours, so it was turned off. It was suspected that TCU#3 was the source 
of the GN 2 leaks, so it was taken off-line and a small rough pump was hooked up to the gas supply line. This was 
continually rough pumped for about a week. The solar array cryopanels this TCU was controlling was allowed to 
drift a little bit more, but was still reasonably maintained within temperature range by strip heaters on the cryopanels. 
It was then decided to open all 3 cryopumps, since there is essentially no Helium. Pressure levels could hold -1.0E06 
torr during Cold Plateau #3. The project then sides with flooding the shrouds to further create a large cryo trap to 
collect all water vapor and nitrogen on the shrouds walls! This now keeps the pressure steadily below lE-6 torr and 
High voltage testing commences anxiously. While checking and demelting ice buildup by the solenoids, the staff 
noticed that the vent valves were closed (probably from previous leak checking) and were manually reopened to 
relieve LN 2 /GN 2 flow through the cryo-flex lines. Pressure levels subsided a little more and pressure spikes disappear. 
HPCA conducts High voltage testing and completes ranging to an operational level of 10,000 volts! 

It was looking good to continue with HV testing, but pressure levels gradually inch up and it was discovered 
that Omega #10 solenoid valve stuck open again and several other supply lines were sent cold as an ice ball enveloped 
several control systems near this solenoid. At this point of the test the option of stopping LN 2 again to these circuits, 
including the scav plates, was not considered, therefore the ice ball had to be melted to gain access to that individual 
solenoid individual solenoid . A software fix for controlling the solenoids was to be implemented, so the CD ACS was 
taken off-line briefly while the new fix was implemented. Nominal HV testing resumed and completed thereafter. 


The next phase was to place the OBS into an eclipse mode. This had not been since the first OBS2 TV test. 
This involved putting TCU#3 back on-line, preheating the OBS (and IS) and then flooding all the cryopanels. The 
shrouds were already flooded from the previous decision a few days earlier. As the cryopanels were flooded, a large 
pressure spike of GN2 was noted. Chamber pressure increased to 1.2E-5 torr, project personnel got nervous, and 
fortunately the pressure levels subside and then magically decreases sharply and reaches -4E-7 torr within minutes! 
Overall chamber and ion gauge pressure levels reach even lower, to -lE-7 torr during the eclipse mode. Figure 15 
shows the dramatic pressure changes during implementation of eclipse settings. 
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Figure 15: Chamber pressure changes going to Eclipse set points 


The main pressure drop was suspected to be from the chamber’s burst disk reseating after the pressure spike. 
This disk is located at the top of the chamber and was not easy to detect if the gasket in it is probably seated. It had 
been a noted leak source in previous tests. Anyways the eclipse mode lasted about 6 hours and then the cryopanels 
were adjusted to conduct the last set of thruster firings. These firings went off nominally, with RGA data being tracked 
and collected, as requested. Ion gauges recorded similar pressure spikes as in previous tests. The project polled the 
OBS and IS team members whether any additional testing was requested now that the pressure levels have been 
corrected. No further testing was requested and the return to ambient was completed without any further issues. 
Figure 16 shows the entire TV profile of OB S3. The chamber pressures were certainly more elevated and there are 
noticeably more pressure spikes in this TV test, when compared to OBSl data (Figure 13). 
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Figure 16: OBS3 Thermal Vacuum pressure profile 


LESSONS LEARNED 

• Complex tests are difficult to capture 100% of every step or procedure needed to complete an uneventful test. 

• Expect problems, consider workarounds and have back plans before testing begins. 

• Consider leak checking with inert gases, like argon, instead of Helium. Perhaps reserve Helium use to confirm a 
leak first detected through use of argon. 

• Pressure leak test TCU circuits before vacuum. Reserve adequate time to complete leak checking of enclosed 
circuits. 

• Conduct precerts and/or dry runs to shake out the entire system. Conduct a mini-cycle to validate operation of 
thermal control systems as they will be conducted with spacecraft testing. Exercise all monitoring equipment to 
validate their functionality beforehand. 

• Utilize multiple ion gauges whenever pressure issues are concerns. 

• Maintain spacecraft operational heaters on until shrouds and other test support panels have been adequately 
warmed up > IOC. 

• Check and re-check the known problems or nuances of the vacuum chamber. Replace with more reliable 
apparatus whenever possible. 

• In complex TV tests, have real-time monitoring, even if it conducted remotely, which allows for catching issues 
in almost real time. This allows for corrective actions to be completed in a timely fashion. 


